The present invention relates to an image processing apparatus, an image processing method and a transmission medium, and more particularly to an image processing apparatus, an image processing method and a transmission medium which are used for calculating a distance to a target point with a stereo method.
The so-called stereo method is a technique to measure a distance to a target point based on the principle of triangular surveying by using images picked up from a plurality of visual points.
FIG. 1 shows an example of configuration of a conventional image processing apparatus for measuring a distance to a target point with the stereo method.
In FIG. 1, memories 901-1 to 901-5 store one-frame images picked up by cameras 1 to 5, respectively, and image data are read out of the memories to be output in predetermined sequence. Of the cameras 1 to 5, the camera 1, for example, is set as a reference camera. An image of the reference camera is compared with each of the other cameras (compared cameras) to perform a matching process, and a parallax is calculated in accordance with a result of the matching process.
An SAD (Sum of Absolute Difference) circuit 902 calculates an absolute value of difference for each pixel value between the reference image output from the memory 901-1 and each of the images output from the other memories 901-2 to 901-5. For example, the SAD circuit 902 calculates absolute values of differences for each pixel value between the memories 901-1 and 901-2, between the memories 901-1 and 901-3, between the memories 901-1 and 901-4, and between the memories 901-1 and 901-5, and then outputs the absolute values.
An SSAD (Sum of SAD) circuit 903 outputs results of an inter-camera block matching process performed based on the absolute values of differences in pixel value between the cameras.
A minimum value detecting portion 904 detects a minimum value from among the results output from the SSAD circuit 903.
A second-order portion 905 performs approximation (interpolation) for the minimum value output from the minimum value detecting portion 904 and other values thereabout by using a second-order function to calculate a minimum value with higher accuracy.
A memory 906 stores the minimum value output from the second-order portion 905.
The operation of the conventional image processing apparatus thus constructed will be described below.
Assume now that the camera 1 is disposed at the center and the other cameras 2 to 5 are disposed so as to surround the camera 1. An image output from the camera 1 is stored as a reference image in the memory 901-1. Images output from the other cameras 2 to 5 are stored respectively in the memories 901-2 to 901-5.
The SAD circuit 902 creates a pair of the reference image output from the memory 901-1 and each of compared images output from the other memories 901-2 to 901-5, and then calculates and outputs an absolute value of difference for each pixel value between both the images of each pair.
The SSAD circuit 903 performs a matching process for a predetermined pixel block (e.g., a 5xc3x975 block) based on the absolute values of differences in pixel value for each pair of the images output from the SAD circuit 902. In other words, the SSAD circuit 903 calculates a difference value in units of a pixel block, and supplies calculated results to the minimum value detecting portion 904.
The minimum value detecting portion 904 detects a minimum value from among the difference values output from the SSAD circuit 903, and supplies the detected minimum value to the second-order portion 905.
The second-order portion 905 performs approximation (interpolation) for the minimum value output from the minimum value detecting portion 904 and other data of two points thereabout by using a second-order function to create and output a minimum value with higher accuracy.
The memory 906 stores data output from the second-order portion 905.
In processing based on the stereo method, it is required to solve the problem of corresponding points, i.e., determining which pixel of a compared image each pixel of a reference image corresponds. The problem of corresponding points is generally solved by a matching process. Stated otherwise, by comparing the reference image and the compared image, portions showing a maximum matching (similarity) therebetween are determined as corresponding points and correlated to each other.
When searching the corresponding points based on the matching process, a search area is decided beforehand by previous calibration. Theoretically the search area should exist along a straight line called an Epipolar line, but practically it exists along a curved line due to lens aberrations, etc. in many cases.
To overcome such a problem, it is thought to list all search points and store them in a lookup table beforehand. Even in the case of using the lookup table thus created, an intensity value of each pixel may differ between cameras of differences in the kinds of lenses and diaphragms used, resulting in a problem that the matching process cannot be precisely performed.
With a view of overcoming the above problem, processing to compress an input image into 4 bits and to emphasize a portion, where an intensity value changes to a large extent, for absorbing a difference of the intensity value has been proposed (see xe2x80x9cA Stereo Machine for Video-rate Dense Depth Mapping and Its New Applicationsxe2x80x9d, Takeo Kanade, Atsushi Yoshida, Kazuo Oda, Horoshi Kano and Masaya Tanaka, Proceedings of 15th Computer Vision and Pattern Recognition Conference (CVPR), Jun. 18-20, 1996, San Francisco).
With this proposed method, however, a fine intensity difference of an original image is ignored and cannot be reflected on a result of the matching process. Hence there has been a problem that a correct result of the matching process is obtained in a portion where changes of the intensity value are large, while a correct result of the matching process cannot be obtained in a portion where changes of the intensity value are small.
Further, because it is desired that distance data resulting from the matching process be obtained with as high an accuracy as possible, the conventional image processing apparatus mentioned above is designed to perform interpolation for minimum value data, which results from searching the corresponding point, by using a second-order curve and obtain data with higher accuracy.
Executing the interpolation however requires a division process which entails a relatively high computing cost. To reduce a circuit scale needed, therefore, the interpolation has been hitherto executed only upon a minimum value among results obtained from the block matching process (i.e., output of the minimum value detecting portion 904). This method can reduce a hardware scale, but causes a delay in the overall process because the interpolation is required to be executed after the minimum value has been output. Accordingly, this method has caused difficulty in speeding up the processing.
Further, it is important in the stereo method to improve accuracy of a distance to the target point. From this point of view, a stereo processing using a plurality of cameras and a plurality of base line lengths (distances between the cameras) has been proposed, by way of example (see xe2x80x9cStereo Matching Utilizing a Plurality of Base Line Lengthsxe2x80x9d, Journal of The Institute of Electronics, Information and Communication Engineers (Japan), D-II, Vol. J75-D-II No. 8, pp. 1317-1327, August 1992).
With this proposed method, an error in matching can be reduced and distance accuracy can be improved as a result of using a plurality of cameras in a combined manner.
In the proposed method, however, the stereo processing is executed in accordance with one combination of the plural cameras (i.e., by using all the cameras in one combination). Therefore, the proposed method has had difficulty in dealing with the problem of occlusion and hence a disadvantage that distance accuracy is lowered.
With the above in mind, it has also been proposed to prepare several combinations of cameras and to select the best one of those combinations.
That method can improve distance accuracy, but has accompanied another problem that when selecting the best one of the several combinations, if search areas are different between cameras, it is very intricate to deal with such a case.
Further, when implementing the above method with hardware, there has occurred a problem of requiring a longer processing time. Specifically, because an amount of data to be processed is different for each camera, the step of selecting the best combination must wait, until processing for all the combinations has been completed.
The present invention has been made in view of the state of art described above, and its object is to enable a matching process to be executed precisely and at a high speed. Another object is to improve distance accuracy and to simplify processing in the case of searching the corresponding points using a plurality of cameras based on a stereo method.
To achieve the above objects, according to a first aspect of the present invention, there is provided an image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, the apparatus comprising a first input device for receiving, as a reference image, an image supplied from one of the plurality of image pickup devices; a second input device for receiving, as a compared image, an image supplied from another image pickup device; a matching device for performing a matching process between the reference image and the compared image applied from the first and second input devices; an interpolating device for performing interpolation for data obtained by the matching device; and a detecting device for detecting a minimum value from among data resulting from the interpolation performed by the interpolating device.
According to a second aspect of the present invention, there is provided an image processing method for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, the method compromising: receiving, as a reference image, an image supplied from one of the plurality of image pickup devices; receiving, as a compared image, an image supplied from another image pickup device; performing a matching process between the reference image and the compared image; performing interpolation for data obtained from the matching step; and detecting a minimum value from among data resulting from the interpolating step.
According to a third aspect of the present invention, there is provided a transmission medium for transmitting a computer program used in an image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, the medium transmitting a program comprising: receiving, as a reference image, an image supplied from one of the plurality of image pickup devices; receiving, as a compared image, an image supplied from another image pickup device; performing a matching process between the reference image and the compared image; performing interpolation for data obtained by the matching step; and detecting a minimum value from among data resulting from the interpolating step.
According to a fourth aspect of the present invention, there is provided an image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, the apparatus comprising: a first storage device for storing an image output from each of the plurality of image pickup devices; a second storage device for storing a sequence of reading out pixel data constituting the image stored in the first storage unit; a reading unit for reading out the pixel data stored in the first storage unit in accordance with the sequence stored in the second storage unit; and a converting unit for converting gradation of the pixel data read out by the reading unit.
According to a fifth aspect of the present invention, there is provided an image processing method for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, the method comprising: storing an image output from each of the plurality of image pickup devices; reading out pixel data, which constitute the image stored in the storing step, in accordance with a reading sequence stored beforehand; and converting gradation of the pixel data read out in the reading step.
According to a sixth aspect of the present invention, there is provided a transmission medium for transmitting a computer program used in an image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, the medium transmitting a program comprising: storing an image output from each of the plurality of image pickup devices; reading out pixel data, which constitute the image stored in the storing step, in accordance with a reading sequence stored beforehand; and converting gradation of the pixel data read out in the reading step.
According to a seventh aspect of the present invention, there is provided an image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, the apparatus comprising: a first input unit for receiving, as a reference image, an image supplied from one of the plurality of image pickup devices; a second input unit for receiving, as a compared image, an image supplied from another image pickup device; a searching unit for searching an area, which corresponds to a predetermined area of the reference image applied from the first input unit, from the compared image applied from the second input unit by a matching process; a detecting unit for detecting if the area searched by the searching unit is outside a region of the compared image; and a control unit for controlling the matching process performed by the searching unit in accordance with a result detected by the detecting unit.
According to an eighth aspect of the present invention, there is provided an image processing method for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, the method comprising: receiving, as a reference image, an image supplied from one of the plurality of image pickup devices; receiving, as a compared image, an image supplied from another image pickup device; searching an area, which corresponds to a predetermined area of the reference image from the compared image by a matching process; detecting if the area searched in the searching step is outside a region of the compared image; and controlling the matching process performed in the searching step in accordance with a result detected in the detecting step.
According to a ninth aspect of the present invention, there is provided a transmission medium for transmitting a computer program used in an image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, the medium transmitting a program comprising: receiving, as a reference image, an image supplied from one of the plurality of image pickup devices; receiving, as a compared image, an image supplied from another image pickup device; searching an area, which corresponds to a predetermined area of the reference image, from the compared image by a matching process; detecting if the area searched in the searching step is outside a region of the compared image; and controlling the matching process performed in the searching step in accordance with a result detected in the detecting step.